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ABSTRACT 

■ The Chandra X-ray Observatory (CXO), NASA's latest "Great Observatory", was launched on July 23, 1999 and 
' reached its final orbit on August 7, 1999. The CXO is in a highly elliptical orbit, approximately 140,000 km x 10,000 
km, and has a period of approximately 63.5 hours (« 2.65 days). Communication with the CXO nominally consists 
^ ^ of 1-hour contacts spaced 8-hours apart. Thus, once a communication link has been established, it is very important 
^ that the health and safety status of the scientific instruments as well as the Observatory itself be determined as 
ly^ ] quickly as possible. 

In this paper, we focus exclusively on the automated health and safety monitoring scripts developed for the 
Advanced CCD Imaging Spectrometer (ACIS) to use during those 1-hour contacts. ACIS is one of the two focal 
plane instruments on-board the CXO. We present an overview of the real-time ACIS Engineering Data Web Page 
and the alert schemes developed for monitoring the instrument status during each communication contact. A suite of 
("s^ HTML and PERL scripts monitors the instrument hardware house-keeping electronics (i.e., voltages and currents) 
T-H ' and temperatures during each contact. If a particular instrument component is performing either above or below 
pre-established operating parameters, a sequence of email and alert pages are spawned to the Science Operations 
Team of the Chandra X-ray Observatory Center so that the anomaly can be quickly investigated and corrective 
actions taken if necessary. We also briefly discuss the tools used to monitor the real-time science telemetry reported 
by the ACIS flight software. 
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C§ : 1. INTRODUCTION 

^ ■ Just past midnight on July 23, 1999, the space shuttle Columbia lifted-ofF from Cape Canaveral, Florida. In its 
payload bay lay the Chandra X-ray Observatory (CXO), the primary cargo of the STS-93 mission. Just under 8 
■ hours after launch, Chandra was deployed from the space shuttle. However, it would be nearly two weeks later, after 
Ci an Inertial Upper Stage booster "burn" and several "burns" by its own propulsion system, that Chandra would reach 
its final orbit. The CXO is now the third of NASA's "great observatories" in space. 

The CXO's operational orbit has an apogee of approximately 140,000 km and a perigee of nearly 10,000 km, 
with a 28.5° initial inclination. The CXO's highly elliptical orbit, with an orbital period of approximately 2.65 days, 
results in high observing efficiency. Moreover, the fraction of the sky occulted by the Earth is small over most of 
the orbital period, as is the fraction of time when the detector backgrounds are high as the CXO dips into the 
Earth's radiation belts. Consequently, approximately 85% of Chandra's orbit is available for observing. In fact, 
uninterrupted observations lasting as long as 2.3 days are possible.^ 

The CXO carries two focal plane science instruments: the High Resolution Camera (HRC) and the Advanced 
CCD Imaging Spectrometer (ACIS). The Observatory also possesses two objective transmission gratings: a Low 
Energy Transmission Grating (LETG) that is primarily used with the HRC, and the High Energy Transmission 
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Grating (HETG) that is primarily used with the ACIS. In addition to those instruments, Chandra also carries a 
radiation monitor the Electron, Proton, Helium Instrument (EPHIN) partiele deteetor. 

In order to maintain this high level of observing efficiency, a robust monitoring and alert system must be in place 
to verify the health and safety status of not only the instruments themselves, but the whole observatory as well. This 
requirement is a result of the fact that the CXO is not in constant communication with the ground. Communication 
with the CXO is obtained through NASA's Deep Space Network (DSN) of ground-tracking stations. Since the DSN 
supports satellite operations of many different NASA missions, this resource is shared amongst all satellites currently 
in space. 

Chandra's nominal communication schedule consists of 1-hour contacts spaced approximately 8-hours apart. 
Therefore, over the course of one 24-hour period, the Chandra X-ray Observatory Center (CXC) receives telemetry 
from the observatory three times per day for a total of three hours, on average. Of course, during the periods in 
which the ground is not in contact with Chandra, routine science operations continues autonomously on-board the 
spacecraft. Each week, a new set of command loads is generated, reviewed, and uplinked to Chandra for use in 
the following week. These command loads contain all the necessary information for the observatory to execute and 
perform a week's worth of science observations and spacecraft maintenance activities. In a companion paper, ^ we 
review the command load inspection procedure for the ACIS instrument. The science data and observatory health and 
safety telemetry are stored on Chandra's solid state recorders (SSR). During a real-time communication contact, these 
data are telemetered to the ground and transfered to the CXC for data processing (via the Jet Propulsion Laboratory). 
Another paper in this volume^ discusses the monitoring and trends analysis of the SSR data. However, since it can 
take as much as 12-24 hours for the SSR data to be processed, analysed, and hence a putative anomaly found, it is 
vital that the health and safety status of the Observatory be quickly established during each communication pass. 
In addition to telemetering SSR data, "live" or real-time telemetry data from the observatory and all instruments, 
i.e., the current values of all the observatory and instrument house-keeping electronics, are sent immediately to the 
Chandra Operations and Control Center (OCC) during each DSN communication pass. 

In this paper, we focus exclusively on the real-time monitoring of the Advanced CCD Imaging Spectrometer 
(ACIS) on-board the CXO during these one hour communication contacts. In Section 2, we provide a brief synopsis 
of the CXO and its primary focal plane instruments while Section 3 illustrates the data flow from spacecraft to 
monitoring. In Section 4 we discuss the method by which we arrived at our set of health and safety limits. In Section 
5, we present the real-time monitoring schemes developed for the ACIS instrument. Our conclusions follow in Section 
6. 

2. CHANDRA X-RAY OBSERVATORY'S FOCAL PLANE INSTRUMENTS 

The observatory consists of a spacecraft system and a telescope/science-instrument payload. The spacecraft system 
provides mechanical controls, thermal control, electric power, communication/command/data management, and 
pointing and aspect determination. This section briefly describes the two focal plane instruments on-board the 
CXO, the HRC and the ACIS, while the main emphasis in this paper is the real-time health and safety monitoring 
of the ACIS instrument. The AXAF Observatory Cuide'^ and the AXAF Science Instrument Notebook^ contain 
a wealth of information about the CXO and its scientiflc instruments. More in depth discussions of the Chandra 
mission, spacecraft, other instruments and subsystems are presented elsewhere.^''''^'^'^^'^^ 

2.1. High Resolution Camera (HRC) 

The High Resolution Camera, HRC, is a microchannel plate (MCP) instrument. It is comprised of two detector 
elements, a ~ 100 mm square optimized for imaging (HRC-I) and a ~ 20 x 300 mm rectangular device optimized for 

the Low Energy Transmission Grating (LETG) Spectrometer readout (HRC-S). 

The HRC has the highest spatial resolution imaging on Chandra < 0.5 arcsec (FWHM) matching the High 
Resolution Mirror Assembly (HRMA) point spread function most closely. The HRC energy range extends to low 
energies, where the HRMA eflFective area is the greatest. The HRC-I has a large fleld of view (31 aremin on a side) 
and is useful for imaging extended objects such as galaxic^s, supernova remnants, and clusters of galaxies as well 
as resolving sources in a crowded field. The HRC has good time resolution (16 /isec), valuable for the analysis of 
bursts, pulsars, and other time-variable phenomena but has limited energy discrimination, E/AE ~ 1 (< 1 keV). 
The HRC-S is used primarily for readout of the low-energy grating, LETG, for which its large format with many 
pixels gives high spectral resolution (> 1000, 40-60 A) and wide spectral coverage (3 - 160 A). 



2.2. Advanced CCD Imaging Spectrometer (ACIS) 

ACIS is the Advanced CCD Imaging Spectrometer. It is comprised of two arrays of CCDs, one optimized for imaging 
wide fields (ACIS-I; a 2x2 chip array with 16 arcmin on a side), the other optimized for grating spectroscopy and 
for imaging narrow fields (ACIS-S; a 1x6 chip array; 8 arcmin x 48 arcmin). Each array is shaped to follow the 
rc;k;vant focal surface. In conjunction with the HRMA, the ACIS imaging array provides simultaneous time-resolved 
imaging and spectroscopy in the energy range ~ 0.5 - 10.0 keV. When used in conjunction with the High Energy 
Transmission Gratings (HETG), the ACIS spectroscopic array acquires high resolution (up to E/AE = 1000) spectra 
of point sources. "'^^ The CCDs had an intrinsic energy resolution (E/AE) which varied from '--^5 to ^50 across the 
energy range. However, due to radiation damage early in the mission, this has been significantly compromised. ^^'^^ 
Nevertheless, approximately 90% of all Chandra science observation time available during the first four years of the 
mission has utilized the ACIS instrument. 

ACIS employs two varieties of CCD chips. Eight of the 10 chips arc "front-side" (or PI) illuminated. That is, the 
front-side gate structures are facing the incident X-ray beam from the HRMA. Two of the 10 chips (SI and S3) have 
had treatments applied to the back-sides of the chips, removing the insensitive, undcplcted, bulk silicon material 
and leaving only the photo-sensitive depletion region exposed. These "back-side" , or BI chips, are deployed with the 
back side facing the HRMA. BI chips have a substantial improvement in low-energy quantum efficiency as compared 
to the FI chips because no X-rays are lost to the insensitive gate structures but suffer from poorer charge transfer 
inefficiency and poorer spectral resolution (prior to launch). In addition, early analysis from on-orbit data indicate 
that the BI CCDs are more susceptible to "background flares" , which may compromise a measurement, than are FI 
CCDs.^^ These background flares {i.e., rapid increases in detector background) are thought to be a consequence of 
Chandra's radiation environment.^^ 

3. OUTLINE OF DATA FLOW 

As outlined in the Introduction, there are two data streams that are monitored for health and safety issues: (1) data 

received from the solid state recorder, and (2) "live" or "real-time" data received directly from the CXO via the 
DSN. In this section, we will briefly outline the data flow associated with the second stream. The former is presented 
elsewhere in these proceedings.^ 

Once a communication link has been established with a ground-tracking station, real-time telemetry begins flowing 
immediately to the CXO Operations and Control Center in Cambridge, MA. The relevant ACIS house-keeping 
telemetry must be extracted from the raw telemetry before any health and safety monitoring may be performed. To 
accomplish this task, a C-I-+ software package called ACORN (A Comprehensive object-ORiented Necessity) was 
written by programmers at the CXC. 

In our implementation, the real-time raw telemetry stream is fed to ACORN via a User Datagram Protocol 
(UDP) port. The raw Chandra telemetry consists of approximately 11,000 mnemonic string identifiers (MSIDs). 
Every spacecraft and instrument sensor, thermistor, meter, etc. arc iclentificKl and tracked via a unique MSID. 
ACORN decodes the real-time telemetry stream and writes MSIDs, values, and spacecraft times to either a tab- 
delimited file or standard output. For our purpose, ACORN-decoded data is written to a tab-delimited file so that 
it may be analyzed. This entire process of real-time raw spacecraft telemetry to output files that can be processed, 
is depicted in Figure 1. 

4. DETERMINING ACIS HEALTH AND SAFETY LIMITS 

In order to monitor the health and safety status of the ACIS instrument, a set of MSIDs as well as a set of limits must 
first be established. In total, there are 65 ACIS MSIDs that are regularly sent by the instrument to the observatory 
for incorporation into the telemetry stream that constitute the ACIS hardware "house-keeping telemetry" . These 
MSIDs consist of components like the camera-body temperature to the ACIS focal plane temperature (please see 
Figure 2). By and large, these are the principle electronic {i.e., voltages and currents) and temperature MSIDs 
reported by ACIS. 

Prior to launch, as well as during the first 2 years of the mission, there were several sets of "limits" used by 
various teams within the CXC to ostensibly suit their own particiilar purpose. The original health and safety limits 
{i.e., pre-launch) were established based on the qualification tests conducted during the instrumental and observatory 
thermal-vacuum tests. Therefore, the first task was to update and streamline the use of various limit sets to one 
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Figure 1. Real-time Data Flow: The pathway by which real-time telemetry from the spacecraft results in ACIS 
monitoring web pages. 



definitive; set. Two years into tlic mission, after sufiicic;nt c;xperience had been gained with instrnment response to 
on-orbit use, various members of the CXC convened to identify a robust set of hmits to check against the real-time 
ACIS telemetry feed. 

It was quickly decided that one set of limits would not be desirable since there are principally two concerns that 
would govern what the limits should be: (1) health and safety concerns, and (2) data quality concerns. Clearly, the 
former is more important than the latter since the former is used to monitor and maintain vital mechanical operation 
whereas the latter serve to indicate possible reduction in data quality. Moreover, the philosophy of the the health 
and safety limits is such that should a "red" limit violation occur, the CXC response will be immediate. Should a 
"yellow" limit occur, further analysis may be performed before any response would be initiated. With this operating 
principle, two sets of limits were created for each MSID, with each set containing caution (color-coded yellow) low 
and high values and warning (color-coded red) low and high values that would be applicable to health and safety 
concerns as well as concerns that would affect data quality. 

Nearly 300 days worth of data from the second year of the mission were analyzed to arrive at the ACIS health and 
safety as well as data quality "yellow" and "red" limits. Limits from the MIT/ ACIS instrument team were compared 
against this 300-day baseline and modified on a case-by-case basis. For some MSIDs, the concept of either a low or 
high "yellow" and/or "red" limit has no meaning and so these MSIDs were assigned a nominal value of "999" to 
indicate such a condition. These limits are presented in Figure 2. Once this definitive set of limits was determined, 
they were implemented by every group within the CXC that monitor the telemetry from the spacecraft. One caveat 
that should be noted is that these MSIDs label telemetered sensor readouts such as the camera-body temperatures 
and the various voltages and currents generated by the ACIS power supply (see figure 2). The ACIS focal-plane 
temperature, FP-TEMP, is not included with the MSID values sent to ACORN but is extracted from the Digital 
Electronics Assembly (DEA) house-keeping packets reported by the ACIS flight software. It is unavailable when the 
flight software is halted or when the DEA house-keeping is disabled. 

5. REAL-TIME MONITORING OF THE ACIS INSTRUMENT 

There are two different ways in which the health and safety of the ACIS instrument is monitored. The first is by 
monitoring and limit-checking the 65 ACIS house-keeping MSIDs discussed in the preceding section. The second 
method is by monitoring the ACIS science data packets produced by the ACIS flight software system. The former 
is discussed in Section 5.1 while the latter is discussed in Section 5.2. The principle difference between these two 
methods is that the former monitors telemetry reported by the hardware components on ACIS, whereas the latter 
monitors the science telemetry reported by the ACIS flight software. 

5.1. Monitoring the ACIS House- keeping Telemetry 

With the ACIS health and safety limits as well as access to real-time telemetry to perform limit-checking established, 
we now outline the process by which the ACIS telemetry is monitored and key ACIS personnel notifled in case of an 
anomaly. 

As stated in Section 3, ACORN produces tab-delimited ASCII files for a set of ACIS MSIDs. ACORN constantly 

monitors a given UDP port for spacecraft telemetry. When the telemetry is flowing, i.e. when the OCC has 
established a communication contact with the CXO, data from these MSIDs are appended to various ASCII files. 
When the OCC is not in contact with the CXO, these ASCII files are not modified and contain data from the last 
communication pass. 

Three PERL scripts were written that facilitate the monitoring process. One PERL script (acis-read.pl) reads 
the ACIS ACORN log files and calculates an average and sigma (based on 10 samples of data) for each MSID that is 
followed. These values and timestamps are then written to another tab-delimited ASCII file. The second PERL script 
(acis-www.pl) reads these log files and produces an HTML file that displays this information for team members to 
inspect while at work, home, or on travel (see http://cxc.harvard.edu/acis). The HTML headers instruct the user's 
browser to ask for a new version of the page once every 150 seconds. This second PERL script also incorporates the 
instrument "red" and "yellow" limits discussed in the preceding section. If a given MSID deviates from its nominal 
value such that it falls within the "red" or "yellow" regime, the value for that MSID is either color-coded "red" or 
"yellow" on the web page. A third PERL script (limitpager_vl-4.pl) also reads the ASCII log files produced by 
acis-read.pl. It compares the average value for each MSID reported against its database of "red" and "yellow" 
limits. If a MSID breaches its nominal value such that it falls within the "red" or "yellow" value span, a pager 
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Figure 2. ACIS Health and Safety MSIDs: This hsting is a compilation of aU the 65 ACIS MSIDs that are 
monitored for health and safety reasons. A description and the corresponding "red" and "yellow" limits of each 
MSID are presented. 



Date: Sun, 4 Aug 2002 18:10:42 -0400 (EDT) 
From: <author> 
To: <distribution list> 
Subject: ACIS LIMIT TRIP! 



ACIS ALERT During Current COMM Pass! 
YELLOW.LO limit tripped 2 times! 
m\D: 1CRAT 

Last Violation Value: -122.19 

Current OBSID: 61094 

Current Altitude / Direction: 53761 / A 

Date and Time: Sun Aug 4 22: 10:40 GMT 2002 

Ctieck tittp://asc.harvard.edu/mta/RT/acis/www/acls-mean.html for latest Info. 



Figure 3. Email Alert Example: An example of what an email alert looks like when the MSID ICRAT breaches its 
"yellow limit" . 

alert (for "red limit" violations) or an email alert (for "yellow limit" violations) is issued by the script to key ACIS 
personnel so that they are immediately notified and so that corrective actions may be taken if required. To reduce 
the probability of false alerts due to either spurious data or a "noisy" communication contact, the data violation 
must persist for at least 3 samples before an alert is dispatched. 

These three PERL scripts arc executed every 5 minutes via a CRON task maintained by a general group UNIX 
account. This way the thorny issue of permission and read/write priviledges is eliminated. When the OCC is in 
communication with the CXO, the web page produced by acis-www.pl is updated with fresh values. When the 
OCC is not in communication with the CXO, acis-www.pl reproduces the last values recorded from the previous 
communication contact. Since the beginning and end timestamp for each MSID value that went into calculating an 
average and sigma for that MSID is also presented on the web page, the reader is able to determine whether the 
displayed value is representative of the current instrument status or whether it is an "old" value from the preceding 
pass. 

This web page, the ACIS Engineering Data Web Page, is a key element in monitoring the health and safety status 
of the ACIS instrument, and is the primary means by which ACIS anomalies may be identified. Since this web page 
is world-accessible, it allows key personnel of the CXC to monitor the instrument status whether they are at their 
office, at home, or even on travel as long as they have internet access. In Section 5.1.1, we provide examples of what 
was discussed in this section. 

5.1.1. Examples 

In determining the health and safety limits, it was expected that certain MSIDs would violate their nominal operating 

range for certain well-known cases. For instance, it is expected that the cold radiators on sides A and B (MSID: 
ICRAT and ICRBT, respectively) would enter its "yellow regime" during perigee transit at certain times of the year 
as the instrument tends to warm-up during this point in the Chandra orbit. Since the "red" and "yellow" limits 
are designed to bracket this MSID's nominal operating temperature range, we expect a violation to occur at perigee 
transit. Please see Figure 3 for an example of what this alert looks like. Nevertheless, since this is a known condition 
and since science operation is suspended when the CXO enters the Van Allen radiation belts, the email alerts that 
would be spawned by limitpager_vl-4.pl due to "yellow" violations by ICRAT and ICBAT are now suppressed when 
Chandra is entering, at, or exiting from perigee transit. 

Secondly, as was mentioned in Section 2.2, the spectral resolution of the front-side CCDs was found to have 
been significantly reduced early in the mission. It is believed that the primary mechanism for this damage is the 



Date: Fri, 19 Apr 2002 10:25:54 -0400 (EDT) 
From: <author> 
To: <distribution list> 
Subject: ACIS LIMIT TRIPl 

ACIS ALERT During Current COMM Pass! 
YELLOW_LO limit tripped 2 times! 
MS\D: IDPiCACU 
Last Vioiation Vaiue: 0.80 

******************************************** 

Current OBSiD: 2953 

Current Aititude / Direction: 135915 / A 

Date and Time: Fri Apr 19 14:25:52 GMT 2002 

Ctiecic tittp://asc.tiarvard.edu/mta/RT/acis/www/acis-mean.html for latest info. 

Figure 4. Email Alert Example: An example of what an email alert looks like when the MSID IDPICACU breaches 
its "yellow limit" . 

result of 100-200 keV protons scattering off Chandra's mirrors and impacting the CCDs while ACIS was in the 
focal plane during perigee transit very early in the mission. ^^'^^'^^ To prevent further damage to the ACIS CCDs, a 
prevention scheme was implemented by the Science Operations Team of the CXC.^* One aspect of this scheme has 
been to limit the orbital proton fluence and flux as measured by the proton monitors on-board the ACE and GOES-8 
satellites. One way in which the low energy proton populations can become highly elevated occurs when the Sun 
unleashes a coronal mass ejection (CME).^^ Once the orbital fluence has reached its threshold value/^ or if one of 
the EPHIN channels exceeds its threshold value/^ Chandra science may be suspended by ground intervention (in 
the case of the former) or autonomously by the spacecraft (in the case of the latter). When this occurs, the ACIS 
front-end processors are powered down and the Digital Processing Assembly's (DPA) input currents decrease such 
that a "yellow" violation occurs for IDPICACU and IDPICBCU. The email alerts generated by this violation serve as 
a confirmation that Chandra has indeed suspended science operations because of such an event. Figure 4 illustrates 
what this email alert looks like when IDPICACU has violated its "yellow" limit. Since the ACIS Engineering Data web 
page makes extensive use of color and includes various images, an example is not included in this paper (although 
the URL is presented in Figures 3 and 4). 

5.1.2. Peripheral Issues 

Lastly, there are some peripheral issues that need to be addressed in order to demonstrate the robustness of the 
health and safety monitoring. The first issue is one of redundancy. If the machine that hosts the CRON task were to 
shut down either because of an electrical power failure or due to problems local to that host, the scripts would not 
run, the web page would not be produced, and the limit-checking would not be performed. To solve this problem, we 
run these PERL scripts and produce the ACIS Engineering Data web page on two different hosts so that we are not 
prone to a single-point failure. One host, which produces and performs the "primary" web page and limit-checking, 
is part of a "mission-critical" suite of SUN work-stations. Should this machine go down for any reason, at any time 
of the day, the Smithsonian Astrophysical Observatory's computer system group is tasked with the responsibility of 
solving whatever problem may afflict the host as soon as possible. The second host, the "back-\ip" , is run on an 
ACIS group SUN work-station. Secondly, two additional PERL scripts have been written that deletes old, obsolete 
ACORN files as well as ensuring that the ACORN process is active on the CPU. If the ACORN process is not found 
on the CPU, another process is launched. Both of these PERL scripts are also a part of the CRON task that runs 
on both hosts. Since these have been the primary ways in which the web page may not have been always up-to-date 
thus far in the mission, we believe we have now developed a scheme of PERL scripts that is robust to a single-point 
failure and is one that can be relied upon to monitor the health and safety of the ACIS at all times. 



ACIS Process Monitor, Wed Jul 31 03:04:10 2002 



OBSID: 2572 



ACIS Monitor 



Bilevels: 01100101 



Sent: 2002-212Z07 : 04 : 10 



VCDU: 127295:104 BEP ; 7d8a8817 Fmt : 2 Seq: 13726 SCET : 2002-212X07:04 



2002-212T07:04:06 



BIAS ERROR THRESHOLD EXCEEDED BY FEP 3 



Serial Command 
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Result Code 



Bytes 
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Run : 3 



Slot: 4 



ID: 002a6034 



Quad: FULL 



Start 2002-211T19:23:( 



Mode: TE/FAINT 



Exp: 3.2 sec 
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Figure 5. ACIS PMON Web Page: The monitor receives Cliandra telemetry from real-time contacts, extracts ACIS 
fields, and writes an HTML file. 



5.2. Monitoring the ACIS Flight Software Science Telemetry 

The second method by which the health and safety status of the ACIS instrument is monitored each and every DSN 
communication pass is via the ACIS Real-Time Process Monitor (PMON). 

The ACIS PMON also receives Chandra telemetry during each real-time contact via a UDP port. A series of 
shell scripts has been written that decommutates the raw telemetry, extracts ACIS science fields, and produces an 
HTML file that displays these data. This HTML file presents information that is relevant to the ACIS science run 
currently in progress. Information such as which CCDs are in use, the number of x-ray "events" found in a given 
exposure frame, the overclock values for each node amplifier on a given CCD, etc., are displayed. A PMON HTML 
example is presented in Figure 5. 

When PMON receives fresh telemetry, it writes an HTML file every 30 seconds. Like the ACIS Engineering Data 
web page, the headers in the PMON HTML page instruct the user's browser to ask for a new version of the page 
once every 30 seconds. However, when not in real-time contact, or in a non-ACIS telemetry format, some of the 
displayed values may be very out-of-date. The user, normally ACIS operations personnel, must inspect the relevant 
time fields to determine whether the data are still useful. As is the case for the scripts used in monitoring the ACIS 
house-keeping electronics and temperatures, a PERL script has been written that checks to see if the PMON process 
is active on the CPU. If it is not, another PMON process is launched. This PERL script is run every 5 mins via 
another CRON task. 

Date: Wed, 31 Jul 2002 03:03:49 -0400 (EDD 
From: <author> 
To: <distribution list> 

PMON error reported by xconuck at Wed Jul 31 03:03:49 EDT 2002 
Spacecraft Time: 2002-2 12T07:03:39 

Error Message: OBSID 2572 BIAS ERROR THRESHOLD EXCEEDED BY FEP_3 
Check http://asc.harvard.edu/acis/RT/acisrt.html for latest info. 

Figure 6. PMON Pager Alert: This is an example of a recent in-flight anomaly determined via the ACIS PMON 
and the pager alert issued to ACIS personnel to announce its discovery. 

Over the course of three years of successful flight operations, there have been three issues that the ACIS Operations 
team (combining personnel from the CXC as well as the MIT/IPI team) have had to solve with regards to issues 
of ACIS science operation. One, on a few seemingly random occassions, the ACIS bias maps have been interleaved 
with ACIS science data. The efi'ect of which is to compromise the science data for that particular CCD and unless 
corrected, the condition will persist. Since this anomaly was diagnosed, changes to the way in which ACIS is 
commanded via the weekly load have been made such that we have not seen a repeat of this anomaly, the cause of 
which is still unknown. The second issue is one where the bias map reports a large number of parity errors. The 
cause of this anomaly is also not known, however, we have not seen this condition since November of 1999 and it may 
have been the result of a single-event upset. The last issue is one in which a fatal error is reported by the back-end 
processor on ACIS. This causes software patches to the ACIS flight software to be lost from memory and requires 
ground intervention by the ACIS Operations team to resolve. Since all three of these anomalies can be determined 
via the ACIS PMON, a shell script was written that spawns a pager alert (since immediate attention is required) if 
one of these anomalies were to occur during a real-time contact . An example of this type of alert is presented in 
Figure 6 and a comment announcing one of these errors can also be seen on the PMON HTML page (the third line 
from the top in Figure 5). 

6. CONCLUSIONS 



The ACIS instrument on-board the Chandra X-ray Observatory has returned some of the most exquisite, dramatic, 
and detailed images of the x-ray universe ever seen. The information returned, and the knowledge gained from 



those ACIS observations, has allowed us to explore another wavelength in rieh detail that will undoubtedly propel 
astronomy forward in this new millenium. However, the continued success of Chandra, and ACIS in particular, is 
dependent upon diligent operation of the spacecraft as well as the prompt notification, and hopefully resolution, of 
any anomalies encountered in-flight. In this paper, we have presented two distinct methods by which the telemetry 
from ACIS during each and every real-time contact is monitored, and in the event of an anomaly, how alerts are 
quickly dispatched to key ACIS personnel. As new anomalies are discovered in-flight, we will endeavour to find ways 
to incorporate their monitoring within our current scheme. After three years of highly successful flight operations, 
we believe we have developed a very robust and reliable suite of PERL and HTML scripts that will help ensure 
ACIS' continued success. 
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